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Foreword 



The US Army Computerized Training System (CTS) represents the 
resultant of a multidisciplinary effort combining the latest advances in mini- 
computer hardware and software, learning theory and instructional strategies. 
Accordingly, it is appropriate that an evaluation of such a system itself be 
highly multifaceted. In this respect, three major evolutionary stages of the 
CTS are addressed: developmental, operational and projectional. Further, 
within each ot" these stages, three major dimensions of effectiveness are con- 
sidered: technical, cost and training. Again, by the same token, each of 
these dimensions will be assayed on three major levels of analysis: micro- 
scopic (subsystems evaluation), macroscopic (systems evaluation), and tele- 
scopic (follow-on evaluation). Likewise, each of these levels will subsume 
their own manifold perspectives. The end product of the evaluation, however, 
will not be merely an expaiiding diffuse analysis of the multiplex aspects of 
CTS, but a synthesis of the findings toward meaningful operational decisions 
concerning its overall feasibility and effectiveness. 




COL, SigC 

Product Manager, CTS 
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Introduction • 

1. Etiology of CTS Evaluation. The mission of the CTS Evaluation Division is to con- 
duct Phase V of the general development plan as provided for in the Product Manager 
Charier and CTS Management Plan (cf. Appendices A, B). As indicated in the general 
development plan, "This phase will be conducted by the Product Manager concurrently 
with Phase IV (CTS Operation) and is concerned with the feasibility and effectiveness of 
the entire system". This evaluation phase, similar to the other four phases, has been 
charted along a time phase schedule (cf . Appendix C). The key activities identified in 
the Time Phase Plan as essential to the evaluation of the CTS are: (1) form prelimin- 
ary evaluation plan; (2) identify necessary evaluation data; (3) finalize evaluation plan; 

(4) program evaluation requirements; (5) conduct test on evaluation data; (6) conduct 
formal evaluation of CTS; and (7) analyze data and draft final evaluation report. The 
information contained in this report relates to items one and two, and will be consider- 
ed preliminary till item three is completed. 

2. Orientation of CTS Evaluation. The CTS will be evaluated from a multidimensional 
point of view. First, from its inception it was decided that CTS would be tested and 
evaluated in an operational Army training milieu. In consonance with the rationale 
underlying the design, development, and operation of CTS, the keynotes to the evalu- 
ation as well will be "pragmatic" and "functional". Thus, emphasis will be placed on 
operational reliability and utility. This approach will insure that the evaluation will 
not be misdirected in deriving many "nice" answers to a lot of oblique questions. 
Rather, it will insure that the ultimate evaluation product will represent a realistic 
"in vivo" appraisal of CTS with immediate practical results and implications. 

Secondly, within the pragmatic -functional context, the evaluation will be gen- 
erally conceived to be both expository and demonstrative. An expository view will 
include a thorough description of operational events and problems which occur within 
the Course Development/Operation and the System Operation/Programming Divisions. 
The use of accepted procedures of qualitative^naturalistic observation and quantitative 
statistical descriptive measures will be applicable here. The demonsti ative F^pects 
of the evaluation of CTS will include comparative analyses employing both internal and 
external criteria. The use of systematic assessment and evaluation control proce- 
dures employing classical inferential statistical procedures will be utilized as appro- 
priate. 

Thirdly, within the expository-demonstrative context the evaluation will be con- 
ducted on two contrasting levels: macro and micro. The macro level will address the 
global "Gestalt" aspects of CTS; whereas, the micro level view will address its more 
elemental subsystem facets. 



Lastly, for the unhancoment of thoroughness and objectiyity, the entire evalu- 
ation will be oriented toward an adversarial assessment to include both the pro and 
con aspects of the primary components of CTS (instructional model, courseware 
programs, software progrtims, CTS language, etc.) and any unanticipated by-pro- 
ducis and unforeseen side effects. The net effect of this approach will contribute 
to a goal-free (serendipitous) as well as a goal -based (preplanned) evaluation, 
which will insure against a narrow tunnel vision syndrome. 

3. Perspective of CTS Evaluation. For the salve of adequate perspective, a des- 
cription of how the CTS evaluation relates to the entii'e project is presented in 
Figure 1. An overview of the evaluation areas and their relationship to the CTS 
Organization (Figures 2,3) will be discussed in sections 4 and 5 below. As indi- 
cated in Figure 1, the orientation of the evaluation design, similar to the course 
and system design, is contingent upon the general requirements defined for CTS in 
its iMantigement Plan (cf. Appendix B), Subsequent to the design and development of 
the CTS courseware, system hardware/software and evaluation plan, the prototype 
system run will be conducted. The system run will be subdivided into two stages: 
trial and operational. In the trial stage (also called the formative period), the 
courseware and computer software will be evaluated for their internal consistency 
and interactive integrity; and, during the operational phase (also called summative 
period), the integral CTS package will be evaluated for its operational reliability 
and external validity against existing baseline control information. The resultant 
of the multidimensional analyses of CTS are projected to define three courses of 
action: full acceptance and implementation of the final courseware and software 
products, minor or major redesign of the same with subsequent implementation, 

or suspend judgement till the state of the art demonstrates a proper course of 
action. It should be noted that contemporaneous with the formative and summative 
evaluation, other essential aspects of CTS will be assessed. These include course 
development/operation, system hardware/software, and costing of the entire CTS. 

4. Scope of CTS Evaluation. Operating within the scope of the CTS Charter and 
Management Plan the evaluation of CTS is intended to be both intensive and compre- 
hensive. In this context, the depth and breadth of the CTS evaluation will be dir- 
ected toward, and limited by, the US Army's stated evaluation objectives of testing 
the feasibility and effectiveness of CTS (ef. CTS Management Plan: Appendix B). 
On the basis of these two global objectives, five basic subareas have been identified 
for evaluation: hardware, software, cost, courseware and student . For the sake 
of expediency, and by the logic of their relationships, these five subareas will be 
consolidated into three broad evaluation areas which are: System Effectiveness, 
Training Effectiveness and Cost. Effectiveness (cf. Figure 2), 

The thrust of the three pronged evaluation plan will be based on the assess- 
ment dimensions, outlined in section 2 above, as appropriate: pragmatic -functional, 
expository-demonstrative, and micro-macro. For example, the technical effec- 
tiveness of the CTS will address the reliability and utility of the hardware and 
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software subsystcMus and systcMii as a whole (micro and macro views). The strengths 
and wciilvnesses of the system components will include an assessment of their follow- 
on operational system implications (pragmatic view). Performance characteristics 
will include flexibility and extendal^ility of the hardware/software and operating/main- 
tenance characteristics oi' the component subsystems (exiX)Sitory view). As appr». * 
priate, input to these performance characteristics will be drawn from the full ran^e 
of potential consumers: systems operation and course development personnel, in- 
structors, and students via their respective perceptions of the system's components 
(demonstrative view). 

In regards to training effectiveness, the operational reliability and validity of 
both student performance and course efficiency will be examined. Student perform- 
ance broadly includes course achievement as measured by means of written and per- 
formance tests specifically designed to measure the skills and knowledges attained 
(pragmatic view) in each of three Army technical training courses assigned for CTS 
study purposes. Also included will be measures of student attrition, time to com- 
plete training and attitudes toward CTS (expository view). Course efficiency will be 
assayed on 2 levels: formative evaluation (micro-view) utilizing preestablished 
criteria of course excellence; and; summative evaluation (macro-view) via empirical 
testing of CTS courseware against courseware utilizing the same objectives but 
developed along a different instructional method (demonstrative view). These two 
assessments constitute thf .internal and external quality control procedures for 
supporting the optimum dex^elopment of CTS, and will further provide prototype 
benchmark norms for future CTS development. 

Thirdly, the cost analyses will attend to cost budgeting, effectiveness and 
benefits that accrue to the CTS. A multidimensional assessment of CTS costing 
will include type of cost (capital/developmental/operating); purpose of cost (direct/ 
indlrect/noninstructional) and area of cost (subsystems and areas). The scope of 
this cost assessment will range from item costs (micro) to total system cost (macro). 
It is anticipated that the derived cost model will enable acquisition of the following 
required cost information amenable for management information and decision making 
purposes: descriptive costs (budget/cost accounting); predictive costs (cost analyses/ 
forecasting); and, comparative costs (cost effectiveness/benefits). Here, the 
assessment encompasses both expository and demonstrative costs. Finally, it is to 
be noted that the underlying theme of the cost analysis will be the educational bene- 
fits accrued by the costing levels (pragmatic -functiona!). 

The joint findings of the alcove three evaluation areas will contribute to the 
determination of the overall ieasibility and effectiveness of the CTS, This objective 
will also be viewed from three separate aspects: developmental, operational and 
projectional , An outline of the specific objectives apropos to the three evaluation 
areas and a detailed description of each of the areas is contained in the succeeding 
section below and chapters to follow. 



4 




ERLC 



5 



5. Objectives of CTS Evaluation . In koepiiiy with the scope of the CTS Project (uit- 
linod above, the S|)eeinc objectives of the CTS evaluation can be ehissilu»cl into the 
three broad areas of lechnieal, cost ami training effectiveness. The rehitions)u|)s 
of these areas to the overall C TS organisation network is presented in Figun* The 
underlying intention of the evaluation objectives is to provide, with demonstration 
wlien appropriate, a [)anoramic "snapshot" of the findings relating lo the de\'eh)p- 
nient and operational implementation ol CTS for each of tlvree Army technical courses. 
As indicated in Figure ;J, this "snapshot" will be pi-agmatically based upon tlie pro- 
cesses and products v{ the Course Development/operation and the System Operation/ 
Programming Divisions. Thus, the evaluation domain will encompass the ganiut of 
the organizational and functional aspects of these two CTS com|X)nent divisions as 
considered i'rom a t( chaical, cost and training effectiveness |.>oini of view. An out- 
line of the evaluation objectives are presented below. 



Outline of CTS Evaluation Objectives 

I Technical Effectiveness of CTS 

, Required technical cx[)ertise/training of personnel 

. System architecture characteristics/problems (ha)'dware) 

. System i)rogramming characteristics/problems (software) 

. CAI/CMI processing requirements/problems (interactive/batch) 

, System otx?ration/maintenance (procedures /problems) 

. Support of course development/evaluation requirements 

. System 0[XM*ator perceptions of systeni/subS3\stems 

U Cost Effectiveness of CTS 

. Descriptive costs (budget/accoLmting data) 

. Comparative costs (cost effectiveness/benefits) 

. Predictive costs (cost analysis/forecasting) 

Within these broad areas the following costs will be identified: 

. Type of cost (capital/developmental/opcrating) 
. Purpose of cost (direct/indirect /noninstructional) 
. Area of cost (subsystems/components) 

in Training Effectiveness of CTS 

A. Course Development and Operation 

. Required technical expertise/training of personnel 
\ Course development (time/support/problems) 
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Figure 3 

Relationshif) of Evaluation Areas to CTS Organization 
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. Course operation (procedures/processizig requiremeiUs/problems) 

. Role of personnel (instractors/lP/IPES) * 

. Attitudes toward system (instructors/IP/IPES) * 

B. Formative Evaluation 

. Author feedback requirements (procedures/reports/problems) 
. Data processing support (procedures/problems) 

C. Summative Evaluation 

. Benchmark appraisal of CTSA)aseline data: 

. Completion time 

. Course achievement 

. Course attrition 

. Student attitudes 

IV By -Products/Side Effects 

A. Technical Effectiveness 

. Human factors (radiation/eye fatigue/ambience) 
. Other 

B. Cost Effectiveness 

. Cost benefits 
. Other 

C. Training Effectiveness 

. Course development/operation 
. Other 

V Heuristic Aspects of CTS Evaluation 
A. Technical Considerations 

. C PU/media/language 

« Interactive/batch processing (CAI/CMI) 

. System operation 

. Other 



♦Cf. Glossary: Appendix E. 
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B. Course Development /Operation Considerations 



. Strategies 
. Operation 

. Contingency management 
. Other 

VI Summary /Conclusions/Recommendations 

6. Schedule of Evaluation Events. The following schedule contains the projected 
completion dates for the major evaluation events. 



Schedule of CTS Evaluation Events 



Evaluation Events 
o Preliminary Evaluation Plan Submission 

. Technical Effectiveness Plan 

• Cost Effectiveness Plan 

. Training Effectiveness Plan 

. Course Development/Operation* 
. Formative Evaluation Plan* 
. Summative Evaluation Plan* 

o External Consultant Group 

. Selection of Consultants 
. Contracting of Consultants 

• Initial Meeting (Full Board) 

. Follow-up Meetings (Team/Individual) 
. Final Meeting (Full Board) 

o Identify Necessary Evaluation Data* 

o Develop Measurement Techniques (questionnaires/ 

procedures) 

o Program Evaluation Requirements* 
.0 Baseline Data Collection 



Projected 
Completion Date 

9/15/73 

8/15/73 
8/15/73 
8/01/73 

8/01/73 
1/01/73 
3/01/73 



10/15/73 
2/15/74 
3/15/74 
( Variable) 
8/15/76 

12/15/73 



8/01/74 
3/01/75 
5/01/75 
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Schedule of CTS Evaluation Events (Cont) 



Evaluation Events 



Projected 
Completion Date 



o Conduct Interim Evaluation 



1/01/76 



o Analyze Data/Draft Final Evaluation Report 



8/30/76 



The above completion date schedule is necessarily tentative, contingent upon de- 
velopments within the Course Development/Operation and System Operation/Pro- 
gramming Divisions. 

Prior to the final evaluation report, it is anticipated that a series of interim 
reports will be produced not only by the Evaluation Division but the other CTS 
divisions as well. Furthermore, administrative and historical events of interest 
occurring at the management level during the span of the CTS Project will be re- 
ported via periodic progress reports and other publications. In this perspective, 
the CTS evaluation is viewed as only one of several reporting functions of the CTS 
Product Manager's Office. 

7. Consultant Team. In order to obtain added professional expertise an external 
consultant board will be established. This boai^d will be comprised of 3 teams 
representing the major evaluation areas of technical, cost and training effective- 
ness, consisting of eminently qualified individuals drawn from both the civilian and 
military community. Each team will consist of a chairman and approximately four 
panel members all of whom will exercise a consultant capacity albeit on different 
levels. 

A tentative selection of the consultant members was accomplished on 15 October 
1973. The administrative details of contracting for the consultants is expected to 
take several months. For this, and other management reasons, the initial meeting 
with the consultants is planned for approximately March 1974. It is anticipated that 
both the initial and final meeting will be an appropriate time to have all the consul - 
ants rendezvous together at CTS, Fort Monmouth, N. J. At the initial meeting 
relevant methodological matters will be addressed. These will include: the intro- 
duction of consultants and review of their roles and functions; an orientation of CTS 
describing its nature and status; a tour of the CTS facilities including a demonstra- 
tion of the CTS hardware/software system; and a review of the preliminary evalu- 
ation plan. The final itbecting, of course, will consist of a wrap up of the substan- 
tive evaluation events. 

The frequency and mode of interim interactions with the consultants will be left 
variable, to be determined by significant developments and the nature of events 
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occurring within llie CTS Kvnluation I)i\'isi()ii. Thus, ;is ;i iiiiiiiinuiii :imm';i.u'(\ il 
expected that the respective consultants will he (Miij,a«;(Hl on - In I other orea.sions 
during the span of the evahiation depending.; on ( lie oxiuonries ol l ho situalinn, Like - 
wise, the quorum (l^y board, team, indix idual) and inaniior oC onj.;a|;enuMil CTS. 
by proxy) of consultants will l)e ascertained h> the h)}.^ic ol i ho ^ivon ciiHaunslar.oos, 
Il is anticipated that tlie consultants will he oniployod on a toani and indi\i(hial l)asis 
by proxy the major pi'0(XJrlion of the time. ImiII hoard and in(U\i(hjal le\rl consult- 
ant meetings will be held on site as tlie situation w;ii-rants. They will pi c>\ ide au>'T~ 
iary sup|X)rt as needed during the valuation of CTS. 



Technic^ai Kffecl iveni'Ss 

In evaluating the technical aspects of th<' CTS j)i'ototypo, it is mH•c^^.:I v t<» (U 
\'elop data concerning the technical operation of the prototype iiardware, soII\\:m c 
and their interaction as a system.* The data thus (io\('h)pod will providi i sound 
basis for recommendations by the Product Manager (PM) to the Dei^^irtnuiu oi tin 
Army concerning the adequacy of the prototype' tiU'hnieal system, o|H."rat; iKd 'eoi' 
figurations, and areas requiring fuilher reseaicli und development. 

I. Methodology. The technical evaluation will be based on the li^s (ermiiril pr'»! ' - 
type from which estimates for the mean syslcMn (ie. , ;■)(.)() terminals) will be eah 
lated. The technical evaluation will consist of a deseri|)tive summation (d hardwa?^'\' 
software characteristics, and then a detailed evaluation of the CTS pi-cjlolvpe t( < \\ 
nical- system and its individual components as they affect the author/student /in- 
structor relationship. Due to the emphasis on us(»r/system int eraci iori, then 
several performance criteria usually examined in a system (^valuation which m:iv 
not be addressed in this report. The prototype system will be exandned, fuMP. « 
technical viewpoint, by mode of operation (Intei'aeti ve Pi'oeessing/Iiateli Procer.o- 
ing) and area of operation (CAI, GDI, CMI), where possible, rather thaf. Irehi--' ' 
elements (hardware/software/system), ** 

Due to the uniqueness of the CTS prototype, no standard l)enchmaiK piori.- 
exist which can be utilized to evaluate the overall system. Vov this reason, e . 
objective of this evaluation, if feasible and practical, is to establish such a bencn- 
mark program which can be utilized for the operational system. If any Army 
Standard Programming Languages (ASPL) are utili/ed, then existing bcnehmai-K 
programs can be used to check out the efficiency ot the CTS prototype in these* 
areas. 



U. S. Ai-^xny Signal Center and School, "Ivlulti-Mini Computei* Ti'aining System.'' 
Computerized Training System Specification Nutnl)er S-1 jJo-Tj! which initiated a 
Request for Proposal (RFP) by the Product Manc'iger, April 11)72, 
**Cf. Glossary: Appendix E. 

c 



The evaluation will t'oeus, primarily, upon the functions of the student, the 
system and their interaction. The evaluation will include the functions of the instruc- 
tor, instructional programnuM-, administration and required batch processing. Where 
possible, technical data will be identified as occurring during periods of predomin- 
ately CAI, GDI, or CMI operation. 

2. Instruments for Data Collection. Data collection methods to be used are: 

a. Observations. This will include, primarily, personal experiences, opinions, 
and observations by the PM, his staff, the evaluation team, their consultants, and 
the system/courseware designers. This data will be subjective and may lead to 
further quantitative data collection. 

b. Questionnaires, These will be answered by all operational users of the CTS 
system (student/instructor/instructional programmer/entry specialist/systems pro- 
grammer/administrator) on a periodic basis. A typical period might be at the be- 
ginning and end of a course for students and every three months for permanent party. 
The questionnaires will be directed at collecting data on human factors, how the user 
feels the system is reacting to his needs, and suggested improvements. 

c. Internal System Routines (On-Line), This will constitute the benchmark pro- 
gram to determine actual system and individual component efficiency, utilization, 
adequacy and functional reliability. This program will be developed, if feasible, by 
the evaluation team in conjunction with the Systems Programming and Operation 
Division, CTS. 

d. Interviews. Key personnel will be interviewed on a periodic basis to solicit 
opinions, answers to questions, clarifications, and any other data concerning tech- 
nical performance. Personnel to be interviewed are the PM, his staff, the system/ 
courseware designers, and selected authors /students/instructors/instructional pro- 
grammers/entry specialists/ system programmei's/administrators. 

e. Student P9rformance. Data will be maintained on student performance as 
this remains the ultimate test of technical adequacy. This data will be examined 

to determine areas where the technical capabilities or capacity of the system caused 
identifiable variances in student performance, 

:i. Data Items and Collection Methodology, 

I Interactive Mode: 

A. User: 

(a) Human Factors : 
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(I) Dues roadini^' of the tei'minal screen cause eyestrain? 



() Method : The data for this point will he eoUeeted mainly by questionnaires 
from the users. It will allow the PM to determine whether the CTS terminal is the 
one which should be recommended. 

(2) Forgetting unfamiliarity with the keyboard, is the terminal com- 
fortable to use ? 

o Method : (Same as 1). 

(3) Are images on the primary and secondary device clear? 
o Method: (Same as 1). 

(4) Is the carrel design durable and conducive to learning? 

o Method: (Same as 3). The students opinion of this data item is very im- 
portant. He must remain in this area for approximately six hours per day as long 
as it takes him to compleLe the course. Therefore, if the carrel arrangement is 
not comturtiU^le, it should be designed or reconfigured. This will probably be a con- 
tinual item of concern, not only for the prototype, but for each operational system. 
It is yet to be determined whether one configuration would be acceptable for all 
courses, however, it is felt that each course will require its own configuration. 

( .">) Do the carrel nnd lei-minal designs satisfy existing safety standards 
concerning radiation, electrical and mechanical hazards? 

o .^jj^hod: This data will be collected by technical inspection and measurement 
of radiaxion levels, etc, by knowledgeable personnel qualified in these areas. Their 
z'epoits will be included in the evaluation. 

(b) Primar y Devic e: 

(6) What is the time (avg, high, low) to fill the screen with pure textual 

material? 

jJ?i!lL^iL- ^'^^ ^'^^-'"^ ^^^^ this item will be mainly gathered by on-line programs; 
hovve\er, it will also be a (juestion posed on questionnaires to ascertain how people 
believe ihe systen: is ope\ating. The data for this item will be gathered on a sampl- 
ing basis Diid stored in nK;h ix form. The timing for this item will begin when the 
textual maierlal pulled out of queue and sent down the communication channel 
and end when the last element of the chain has been reproduced on the screen. This 
will provide data as to w.hether the system is fulfilling the performance requirements 
desired, and also provide data lor use in simulations of larger systems. 
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(7) Wliat is the time (avg, high, low) to fill the screen with pure graphi- 
cal display? 

o Method: (Same as 6) 

(8) \Vliat is the^|ime (avg, high, low) to fill the screen with a combin- 
ation of text and graphics ? 

o Method ; (Same as 6) 

/ 

(c) Secondftry Device ; ' 

(9) What capabilities anc^ features are included in the secondary device 
such as audio/visual tape, microfiche, etCo ? 

o Method ; This will be by observation and is for general purpose use. 

(10) Is the secondary device convenient, i.e., does the us^p^ilave to 
load tapes , etc . ? \ 

o Method: (Same as 9) ^ 

(11) Is the secondary device computer controlled? 
o Method: (Same as 9) 

(d) Utilization : 

(12) What is the percentage of utilization of the terminal subsystem in 
CAImode? In GDI mode? In CM! mode? 

o Method : An on-line clock routine will be utilized for this item« Each ter- 
minal will be timed for the period that it is active, and the mode designated. This 
information will provide guidance for operational systems on just how many ter- 
minals they need per shift and mode of operation « 

(®) .Special Training and Documentation; 

(13) Wliat individual training is required by the user prior to utilizing 
the terminal subsystem? 

o Method: In order for the system to be effective, the user should not have to 
spend considerable time learniag how he is to interact with the system « This 
should take very little time and become somewhat second nature to him^ There- 
fore, data for this item will be gathered mainly through the use of questionnaires. 



(14) Do adequate training programs and documentation exist for daily 
operation of the t erniinal subsystem ? 



a. Author Training Package 

b. Proctor Training Package 

c. Systems Pi^ogrammer Training Package 

d. IPES Training Package 

e. Administrative Training Package 

f. Student Orientation Training Package 

g. CTS Demonstration Package 



^ Method ; This will be by observation and is for general purpose use. 

(13) What technical expertise is required for instructors, instructional 
and system pre^rammers and entry specialists prior to training? 

o Method : This data will be collected by questionnaire from managerial per- 
sonnel in their respective areas. 



(0 Syste m Response ; 

{!(]) WTiat is the time (avg, high, low) required for the CPU to re- 



spond to a user query? 

o Method; The data for this item will be obtained purely through on-line pro- 
grams. The timing will begin when the CPU starts to process a query from a 
user and end when the CPU hiis fully responded to the query and released it for 
tran.s mission to the user. This will be accomplished on a sampled basis and will 
be the cumulative average of the data from number 17 below. 



(17) How many of sampled CPU response times fall into the following 



time slJeos ? 



0.0 0»5 milliseconds 



b. 



0.51 



1„ 0 milliseconds 



c. 



1.01 



1. 5 milliseconds 



d. 1.53. 



2.0 milliseconds 
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e. 2.01 - 2.5 milliseconds 



i. 2.51-3.0 milliseconds 

g. 3.01-3.5 milliseconds 

h. 3.51-4.0 milliseconds 

i. above 4.01 milliseconds 

o Method: The data here, gained by on-line programs, will provide a distri- 
bution for CPU response times per user query. This will provide some insight in- 
to the percentage of CPU usage by students, and provide data for future simulations. 
It will also provide the instructional programmer with information as to whether 
the responses they desire are too long, or too short. 

(18) What is the time (avg, high, low) from when a user initiates a query 
until the system has completely displayed an answer? 

o Method: Both on-line program and student questionnaires will be utilized to 
gain data for this item. This will aid the PM to insure that the system is meeting 
its performance requirements, and provide authors with information as to whether 
they are tying the system up too much with the average display, or whether they 
can enlarge their average displays and expected responses. 

B. System: 

(a) Response : 

(19) How large do queues build in the system in CAT mode? In GDI 
mode? InCMImode? 

o Method: The data for this item will be obtained by on-line, programs. The 
information will provide insight as to why system response times are anything 
other than expected values. It will also provide an indication as to whether the 
system is I/O bound. 

(20) Where do queues build in the system in CAI mode? In CDI mode? 
In CMI mode? 

o Method: This will also be obtained by on-line programs. In order to reduce 
queue length and improve system response time, the system programmer must know 
where the queues are building. 



(21) How long does it take to reduce queues in the system in CAI mode? 
In CDI mode ? In CMI mode ? 

o Method ; On-line programs will provide this information. This item, along 
with items 19 and 20, will give the system programmers the information they need 
relative to queuing problems to enable them to take corrective action, if and when 
queues become a problem. 

(b) Utilization ; 

(22) What is the percentage load on each job processor for purely 
course work in CAl mode? In CDI mode? In CMI mode? 

o Method: This data will be obtained strictly by on-line programs. To obtain 
this data, it is planned to sample time slices to see what the load on each processor 
is at that time. This will provide information as to just when additional job pro- 
cessors will have to be added to the system as the number of terminals is increased 
and the number of courses on line increase. 

(23) What is the percentage load on each job processor when both stu- 
dents and authors are on-line in CAI mode? In CDI mode? In CMI mode? 

o Meth od; (Same as 22) 

(24) What is the percentage load on each job processor when just 
authors are on-line in CAI mode? In CDI mode? In CMI mode? 

o Method : (Same as 22). The purpose is to see just what actual load the 
authors place on the system. Here data will be obtained during periods when just 
authors are on-line. 

(25) What is the percentage load on each job processor when students, 
authors and batch processing is taking place in CAl mode? In CDI mode? In CMI 
mode? 

o Method; (Same as 22) 

(26) What is the percentage load on the master processor for the above 

four items ? 

o Method : The data for this item will also be obtained by on-Une programs, 
by sampling time slices. The data from this will be utilized to determine just when 
in the expansion of the system a new master processor will have to be added to the 
system. 



(1^7) How much disc storage docs each course occupy? 

o Method : This iieni will also be obtained by on-line programs. The data 
from this item will [)rovide a good indication as lo just how much disc storage a 
course requires. This will allow the PM to mt\ke recommendations as to the num- 
ber ol" dijc drives needed [)cr course. 

(28) How much core does the operating system utilize? 

o Method: The data for this item will be provided by an on-line program. 
This will indicate how much core storage is available for processing. 

(29) How much core is available for single user programs? 

o Method ; The data for this item will be obtained from an on-line program. 
The core available will be the core that is not required for all other uses. This 
will allow the programmer of single use programs to know just how much core he 
has available for use. 

(30) How much core does each software system utilize? 

o Method: This will be done by on-line routines and will be utilized as infor- 
mation needed before adding to any software system. 



(d) Maintenance : 

(31) Whut is the time (avg, high, low) between failures for each hard- 
ware piece ? 

o Method : The data for this item will be maintained on each piece of equip- 
ment, and will be gathered by observation and on-line programs. The information 
from this item will be utilized in determining the reliability of each piece, and the 
system as a whole. 

(32) What is the time (avg, high, low) for repair of each hardware 

piece? 

o Method : Observation will be utilized in gathering this data. It will be 
utilized to determine reliability of service, and the overall system reliability. 
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(33) What effect does downtime on each haj'dwdre pioee have on the 
overall system capability? 

o Method: CH^servation and questionnaires will be utili/.ed to provide data in- 
put for this item. The data obtained here will be utilized to aasess the critieality 
of each piece of hardware on the ove.rall system. It will also provide information 
as to whether the system can operate, and how efficiently, when a particular piece 
is down. 

(34) What effect does power fluctuation have on the overall system? 

o Method: Observations will be used to gather this data and will determine 
just how stable the system power source must be. 

(e) Software : 

(35) Js the authoring language capable of handling all course presenta- 
tion requirements? 

a. Does the software support a single track, linear, with branching 
instructional model? 

b. Is branching available and based on the student's prior experience 
in the field of study and performance in the course? 

c. Is remedial branching available and based on the student's perform- 
ance record? 

d. Is recycling under computer control provided at skill, training 
objective and lesson level? 

e. Does the student have the option to recall previous material? 

f . Are performance records dynamic and changed under computer con- 
trol based on the student's responses? 

g. Does the system permit a multimedia approach to instruction? 

h. Does the system analyze student performance data under computer 
control and provide feedback to the author for course updating? 

o Method : These items will be obtained by questionnaire. This will be a con- 
tinuous operation to insure that all authoring requirements in course presentation 
are being met. 
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(;J0) Can tlie courseware be easily updated? 



u Method : This will I.e aceomplished by questionnaii^e . It is useful in deter- 
mining time and cost required to miike additions to the courseware. 

(37) Can the authoring language be easily updated? 
o Method: Same as nuinber 30 except tor CAI language. 

(f) Records : 

(38) What are the features of the student users, performance and his- 
tory records and how frequently are they updated? 

o Method : This data will be collected by observation and an on-line program 
to measure frequency of update. 

(39) Is there a graduation date and final grade predictor and what is 
its reliability and validity? 

o Method : This data will be collected by observation. Data concerning pre- 
dicted graduation date and final grade will be documented periodically throughout 
the course and correlated with the actual graduation date and final grade. 

(g) Capability of Expansion : 

(40) Can the capacity and capability of the system be expanded or up- 
graded by additional hardware utilizing the modular concept? 

o Method: The data for this item will be obtained from many other data items. 
This will provide the PM with information as to just how many terminals can be 
active before additional components must be added to the other subsystems, and 
the associated time and cost of such expansion, 

(41) Can the system software be easily upgraded? 

o Method: This will be accomplished by observation and questionnaires. It is 
useful in determining time and cost required to make software additions. 

(h) Efficiency : 

(42) Wliat is the number (avg, high, low^ of machine instructions gen- 
enited per line of coding for each software system? 

o Method : This will be obtained by observation based on the efficiency of in- 
dividual compilers and the system's assembler. This is a measure of software 
Q 'efficiency. 

ERIC 20 



(43) How much thrashing is done by the operating system during 
periods of heavy load? 

o Method : This will be accomplished by the use of on-line progi*ams, and 
will be utilized by system operations personnel. This will indicate just how effec- 
tive the system resource monitor is for the prototype, and how it should be changed 
to increase eu'iciency. 

(44) Does the system become input /output bound? 

o Method: The data for this item will be obtained from several other data 
items. It will provide an indication as to where improvements need to be made 
in the operating system. 

(i) Special Training and Documentation ; 

(45) Do adequate training programs and documentation exist for daily 
operations of the computer system? 

o Method: This will be gathered by observation and questionnaires. 

(46) Do adequate training programs and documentation exist for on- 
sHe maintenance of the computer system? 

o Method ; (Same as 45) 

(47) Is complete documentation available for all hardware systems? 
o Method : (Same as 45) 

(48) Is complete documentation available for all software systems? 
o Method ; (Same as 45) 

(j) Environment : 

(49) Does the system require unusual amounts, types, and reliability 

of power ? 

o Method: (Same as 45) 

(50) Does the system require unusual amounts and location restriction 
of equipment space ? 

o Method : (Same as 45) 
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(51) Does the system require unusual amounts and types of environ- 
mental control? 

o Method; (Same as 45) 

II Batch Processing : 

A, Utility Px'ograms: 

(52) What utility programs are available? 
o Method: (Same as 45) 

(53) What are their features? 
o Method: (Same as 45) 

B. Special Programs: 

(54) What special programs are required? 
o Method ; (Same as 45) 

(55) What are their purposes? 
o Method: (Same as 45) 

(56) What are their features ? 
o Method : (Same as 45) 

(57) When and in what quantity are administrative reports processed? 
o Metho d: (Same as 45) 

C. Special Features: 

(58) Are memory ard file protect options available? 
o Method : (Same as 45) 

(59) What levels of priority are permitted? 
o Method: (Same as 45) 
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(CO) What provision is made for a lesson material security access code? 
o Method ; (Same as 45) 

(61) Wliat debugging facilities are available for each language? 
o Method: (Same as 45) 

(62) Are automatic restart (recovery) procedures available? 
o Method ; (Same as 45) 

(63) What capability exists for the system to continue processing when 
system components fail? 

o Method: (Same as 45) 

The i?ijove list is not all inclusive, but rather represents a foundation for 
further evaluation planning. Details as to how each item will be utilized, or where 
a certain timing operation will begin and end, has not been resolved at this time. 
The final technical evaluation will ultimately depend upon the prototype system 
adopted for use with the CTS project . 

The main thrust of the technical evaluation will be to determine whether the 
CTS prototype satis'fies, in an efficient manner, the technical requirements of the 
author, the managerial needs of the instructor and instructional effectiveness for 
the student. Data items will be collected to indicate how the system actually oper- 
ates and the impressions of the users (the above and the system programmers) as 
to how they think it operates. Together, these approaches will provide a broader 
picture as to the technical suitability of the CTS prototype to the military instruc- 
tional environment. 
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Cost Effectiveness 



In evaluating the CTS Pi^ototype in the area of cost, it is necessary to develop 
a cost model which will provide costing information for management and decision 
making purposes. The cost model must provide data for the preparation of descrip- 
tive costs (budget reports/cost accoimting), predictive costs (cost analysis/ fore- 
casting), and comparative costs (cost effectiveness/benefits).* The cost evaluation 
will be primarily summative in nature although several comparative cost measures 
will be provided to give a broad relationship between CTS cost effectiveness and 
other instructional methods. Finally, the process will be repeated for each course 
of instruction adopted for use with the CTS prototype to provide a range of costs for 
CTS courses with varying modes of lesson presentation. 

The modes of presentation referred to in this report are Computer Assisted 
Instruction (CAI), 'Computer Directed Instruction (CDI), and Computer Managed 
Instruction (CMI). Several definitions of these terms exist with many overlapping 
interpretations; therefore, the following definitions and guidelines will be used in 
relating costs to a particular mode of presentation. 

o CAI mode : is storage of instructional material on-line within the 
computer system and the computer serves as the in- 
structional medium. 

o GDI mode : is a conxbinationof CAI instruction and off-line instruc- 
tion under the direct and immediate control of the computer. 
In this mode the student does not sign off the computer. 

o CMI mode : is the presentation of lesson material via off-line media 
under computer management. In this mode the student 
signs off the computer and leaves the terminal area to 
receive instruction. 

o CMI overhead ; consists of those administrative functions such as lesson 
Assignments, test scoring, resource allocation, remedial 
work and monitoring of student progress which occurs during 
all three of the above modes of presentation. Accordingly, 
costs related to CMI overhead will be included in the cost 
of the above modes of presentation with which they are associated. 

It should be noted at this point that the identification of costs by mode of presentation 
is solely for the purpose of accurately predicting future CTS course costs and not 



^Wilkinson, Gene L. ''Needed: Information For Cost Analysis", Educational 
^ Technology , July 1972. 
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for cost comparison of CAI vs CDI vs CML Conclusions concorniny; such n com- 
parison should be avoided due to the fact that such costs arc identified only ns Ihcy 
can be measured within the CTS environment. No attempt has been made to identify 
the cost of CAI vs CDI vs CIVII as stand alone systems, and it is not the intention of 
this evaluation to do so. 

1. Descriptive Costs* Descriptive costs will categorize cost items for several 
different areas providing a complete view of the CTS prototype cost. Tlie CTS pro- 
totype will be examined in terms of type of cost (capital/developmental/operating), 
purpose of cost (direct/indirect/non-instructional), and area of cost (CAl/CDI/C MI). 

The CTS prototype life-cycle cost will be displayed in a manner similar to figure 4. 

2. Predictive Costs. Predictive costs will include estimates of a complete CTS 
system, i.e. , a mean terminal number of 500 vs 128 for the prototype; yearly oper- 
ational cost for follow -on systems; e^^c^ * on cost of equipment down time; and asso- 
ciated cost for system expansion in upgrading. These costs will be ol:>tained using 
simulation, regression, and other analytical techniques. 

The primary objective for calculating predictive costs is to provide planning 
data concerning implementation of a fully operational CTS at major TRAE>OC train- 
ing centers. Based upon predictive costs provided by the prototype, an estimate 
for life -cycle costs, figure 5, for each system can be made. 

At this time, the extent of an operational CTS as to which training centers, 
which courses, and desired timing is unknown. Accordingly, it is the intention of 
this evaluation to develop a model for determination of CTS cost effectiveness at 
individual TRADOC schools based upon their particular requirements and conditions. 
The methodology and cost model to be utilized for predicting cost effectiveness is 
based upon the model developed and reported in CONARC Task Group Report, 
Computer Assisted Instruction, Vol I, April 1972. This model has been updated 
and revised to accommodate the data associated with the CTS prototype. (Cf . 
Appendix D). 

3. Comparative Costs. Comparative costs will be identified and calculated to pro- 
vide a basis for a broad, general comparison of CTS to Conventional Instruction (CI) 
as will be further discussed in paragraph 6. Several measures of cost effectiveness 
will be included for future use in comparing CTS to specific instructional systems. 
Inherent to the calculation and use of the cost effective measures are the following 
necessary assumptions: 

a. Any Instructional Program used in a comparison with CTS must use courses 
of instruction with similar objectives, content and testing criteria. Calculations 
made for such a comparison must agree in methods of depreciation, system utili- 
zation, etc. 



b. Time savings cost can he niiule use of by timely i'eassi<;nment oi' the student 
once he completes a self-paced course. It' the student is not reassit>ned upon com- 
pletion of the course, this savings cost will be of no value and may aclually be a de- 
triment. 

c. The life of an instructional pi'ogram is :J-5 ye;n\s. 

d. Equipment life is 5-10 years. 

The measures of cost effectiveness to be used foi* eacli course considered may 
include, but are not restricted to, the following: 

o Cost of instruction : The unit of measure is the hourly cost of instruction 

po;* student or 

(1) monthly co st 

student contact hours per month 

The monthly cost includes costs relating to hardware, software, course development 
and administration. Each of these items will include amortized costs, as applicable, 
pertaining to capital investment, physical plant, training, operation, maintenance, 
personnel, etc. 

o Graduation cost: The unit of measure is the cost to graduate a student 
calculated on a monthly basis or 

(2) (hourly cost of instruction) X (course duration (avg) of graduate) 

o Attrition cost : The unit of measure is the cost of instruction expended on 
a failing student calculated on a monthly basis or 

(3) (hourly cost of instruction) X (course duration (avg) of failures) 

o Course cost: The unit of measure is the total cost associated with student 
course termination (graduation and attrition) per month or 

(4) (Graduation cost X ^ Graduates) + (Attrition cost X # failures) 

o Time savings cost : The unit of measure is the difference between gradu- 
ation cost with varied course durrHion (early or late completion for self-paced) and 
graduation cost with fixed course duration or 

(5) (Graduation cost (lockstep)) - (graduation cost (self-paced)) 
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4. Cost Model . The model to collect and document the cost items to be used in 
preparation of descriptive, predictivt? and comparative costs is represented by a 
three dimensional matrix. In this manner, each cost item may be identified im- 
mediately as either a Capital, Developmental, or Operating expenditure in the 
areas of Direct, Indirect, or Noninstruction, and whether the cost is related to 
CAI, CDI or CMI, as applicable. The best representation of this matrix would 
appear as shown in figure G, In this approach, cumulative cost in any one direction 
can be easily calculated. 

5. Data Items. The general headings for the individual cost items to be documented 
are shown below. 

I Acquisition (Nonrecurring): 

A, Hardware Cost: 

(a) Master Processor Subsystem: 

(1) Equipment List 

(2) Site Preparation 

(3) Delivery 

(4) Installation 

(5) Acceptance Testing 

(6) Maintenance (Principle Period) 

(7) Documentation 

(8) Training 

(b) Job Processor Subsystem : 

(Same) 

(c) Terminal Subsystem ; 

(Same) 

(d) Administrative: 

(1) Personnel 
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Type 




Figure 6 
Cost Model For Data Collection 
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a. Salary 

b. Travel 

c. Post Support Activities 

(2) Operating 

(3) Maintenance 

(4) Contractuals 

(5) Facilities 

(6) Miscellaneous 
Software Cost: 

(a) Operating Subsystem : 

(1) Development 

(2) Delivery 

(3) Installation 

(4) Acceptance X^ting 

(5) Maintenance (Principle Period) 

(6 ) Doc ument at ion 

(7) Training 

(b) CMI Subsystem: 

(Same) 

(c) Courseware Subsystem: 

(Same) 
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(d) Evaluation Subsystem : 
(Same) 

(e) Other Required Compilers ; 
(Same) 

(f) Administrciti ve; 

(1) Personnel 

a. Salary 

b . Travel 

c. Post Support Activities 

(2) Operating 

(3) Maintenance 

(4) Contractuals 

(5) Facilities 

(6) Miscellaneous 
Courseware Cost: 

(a) Author_: 

(1) Salary 

(2) Additional Training 

(3) Operating 

(Same) 
(c) Support ; 

(1) Training Aids 
(Same) 



(2) ETV 
(Same) 

(3) Typing 
(Same) 

(4) Other 
(Same) 

(d) Administrative: 

(1) Personnel 

a. Salary 

b. Travel 

c. Post Support Activities 

(2) Operating 

(3) Maintenance 

(4) Contractuals 

(5) Facilities 

(6) Miscellaneous 

II Operating and Maintenance (Recurring): 
A. Hardware: 

(a) Personnel (Supervisors, Operators, Repairmen, etc.) : 

(1) Salary 

(2) Additional Training 

(3) Post Support Activities 

(4) Travel 
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(b) Operating 



(c) Maintenanc e 

B. Software: 
(Same) 

C. System: 
(Same) 

in other: 

A. Upgrading Cost: 

(a) Hardware : 

(b) Software : 

(c) System : 

The above list at this time is representative of categories of individual cost 
- items that must be collected. Further detail of cost items is largely dependent 
upon the systepi (hardware, software and manner of operation), currently being 
developed as the CTS prototype. These and other items will be added and further 
detailed as the system definition becomes finer. 

6. Methodology. The cost attributable to CTS will be provided by the Comptroller 
on a monthly basis. Individual expenditure documents, to include operating, con- 
tractual and other costs, will be maintained at the CTS administrative offices. These 
costs must then be broken down into the areas mentioned above, by type. For pur- 
poses of determining monthly cost, developmental cost will be considered a one- 
time cost and not part of the operating system. Capital cost and courseware cost 
will be amortized using the straight -line method of depreciation on a monthly basis 
and included in monthly cost. Sensitivity of the system to amortization and utiliza- 
tion, figure 7, will be included in the evaluation by providing calculations for equip- 
ment amortization of 5 and 10 years, courseware amortization of 3 and 5 years and 
utilization factors of 6, 12, and 18 hours use per school day. 

Inherent to any comparative evaluation is the establishment of a baseline cost 
to be utilized in the comparison. Two options are available at this time: first, 
collect cost for each course (CI) which v/ill be tested under the CTS prototype and 
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Figure 7 

Sensitivity Analysis of Comparative Costs 
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make a one-to-one comparison; or second, obtain a broad, general cost for Con- 
ventional Instruction and use this as a baseline cost for showing a general, compar- 
ative relationship between CTS and CI. The former is dependent upon what course 
is used with the CTS prototype and whether sufficient data exists from the CI course 
for a meaningful one-to-one comparison. If this option is used, baseline data for 
each course will be collected during a comparable period prior to the CTS operation 
period. The latter is general in nature and provides a broad feeling for CTS vs CI 
cost effectiveness. The general approach for comparative evaluation is currently 
planned anticipating the nonavailability of required data from CI courses for a mean- 
ingful test . 

This is a preliminary approach to a cost evaluation of the CTS prototype. It will 
be updated and refined as the plans for the CTS Prototype development, test ana 
evaluation become more crystallized. 

An initial bibliography is provided in a later section of this report as a general 
overview of the recent literature in the economics of CAI and previous effort by the 
Army leading to the CTS project. 

7. References, (Cf, Bibliography Section: pages 65-66), 
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Training Effectiveness 



The training effectiveness of the CTS Pi^ototype will be evaluated on three 
separate levels: administrative, formative and summative. The administrative 
aspect of training effectiveness pertains to the procedures relating to course develop- 
ment and operation. The formative aspect of training effectiveness pertains to the 
appraisal -debugging-revision phase of course development where course content is 
honed to meet certain predetermined quality specifications , The summative level 
of evaluation pertains to the cumulative effectiveness of a given segment or total 
block/module of course material. The general rationale, procedures and data re- 
quirements for these three evaluation levels are outlined below. 

A. Administrative Evaluation: Course Development /Operation 

1. Course Development. The process of developing lesson material has evolved 
through a pragmatic trial -error approach to Computer Assisted Instruction (CAI) 
lesson preparation. 

The combination of subject matter expert, instructor, author and some func- 
tions of a computer programmer has led to the "one-man'' instructional program- 
mer concept. This individual is an expert in his technical training field as well as 
being an experienced instructor and knowledgeable of some basic language commands 
in programming instructional material. The "one-man" concept of CAI lesson 
development procedures has evolved from several years of user development en- 
vironment rather than a research atmosphere. The pragmatic approach to lesson 
preparation has delineated a number of skills, knowledges, and responsibilities for 
individuals tasked to prepare lesson material. 

The content of a lesson is determined by the terminal behavior desired in the 
student. This terminal behavior is guided by the terminal performance objectives 
derived through systems engineering of training. 

The following paragraphs delineate the setiuence of steps required to prepare 
lesson material for computerized presentation. The evaluation will address these 
topics quantitatively, qualitatively or both, as the event dictates. The aim of the 
course development evaluation will be to present both a structural and functional 
description of the CTS course development process. Further, both positive and 
negative aspects of the entire course development process will be addressed. This 
inclusive and candid approach will insure adequate and reliable final products (3 
courses) for immediate operational use and provide guidelines for developing addi- 
tional Army technical courses. 

I The Instructional Programmer: 

A. Instructional Programmer (IP) Resources: 
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(a) System Engineered Training Objectives: 

(1) Terminal Performance Objectives 

(2) Enabling Objectives 

(3) Skills and Knowledges 

(b) Instructional Model 

(c) Authoring Language with Macros or Subroutines 
Preliminary Planning by IP: 

(a) Research 

(b) Determine Teaching Points 

(c) Outline Lesson Sequence 

(d) Determine Method, Media and Strategy 
Lesson Module Development Done Concurrently: 

(a) Textual Development and Formatting 

(b) Slides and Graphics 

(c) Audio 

(d) Practical Exercises 

(e) Supplemental Material 

(1) Information Sheets 

(2) Operation Sheets 

(3) Equipment Manuals 

(4) Programmed Text Booklets 

(f) Coding 

(g) Training Devices 
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D, Coordination Time with: 

(a) Other Instructional Programmers 

(b) Training Aids 

(c) Instructional Programmer Entry Specialists (IPES) 

(d) Common Basic Electronics Training (COBET) or Other Subject 

Matter Specialists 

(e) System Programmers 
£• Review and Revision Times 

(a) Editing and Debugging(On and Off -Line) 

(b) Lesson Review at Lesson by IP 

(c) Supervisor and Peer Review 

(d) Student Trials 

(e) Modification and Revision after Student Trials and Implementation 

F. Time Spent on Record Maintenance and Updating Documentation 

G. Miscellaneous Time: 

(a) Meetings and Conferences 

(b) Additional Training 

(c) Training Other Personnel 

(d) Leave Time 

H. Miscellaneous Information: 

(a) Diary of Significant Events 

(b) Instructional Model 
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(c) Programming Course (POI) 

(d) Follow-on Projections 

n In-House Support: 

A, Instructional Programmer Entry Specialists (IPES); 

(a) On-Line Entry 

(b) Minor Debugging 

B, Typing Support: 

(a) Supplement Material 

(b) Work Requests 

C, Special ADP Suppoit : 

(a) Special Coding of Course Material 

(b) Student Performance Data for Revision 

in External Support: 

A. Training Aids: 

(a) Training Devices 

(b) Illustration 

(c) Photography 

B. Logistics: 

Supplemental Material Reproduction 

C. WFM-TV: 

Short Video Tapes 
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D. Subject Matter Specialist (Other) 



E. Miscellaneous: 

(a) • Private Contractors 

(b) Consultants 

(c) USASCS Staff 

2. Data Collectioo:Procedures/Instruments (Course Development). The following 
data collection procedures/instruments will be utilized in the evaluation of CTS 
course development. 

a. Off-Line. In order to expedite the collection of both quantitative and 
qualitative information regarding the basic processes (and problems encountered) 
of course development a two page form was devised. This form pertains to Instruc- 
tional Programmer Time (Figure 8) and In-House/Extemal Support Time and Cost 
(Figure 9). The agent responsible for logging the rv^cessary information will be the 
instructional programmer. The frequency of his entries as an instructional program- 
mer will be daily; however, entries for in-house/external support time/costs will be 
periodic according to the circumstances relating to each lesson. 

b. On -Line. Besides off-line data collection, reaMime data will also be 
collected on-line pertaining to the instructional programmer and entry specialist. 
It should be noted that while the instructional programmer focuses his attention to 
authoring (primarily off-line), testing, editing, and debugging (on-line), the entry 
specialist is concerned with entering the instructional programmer's work inter- 
actively on-line. Data collection procedures for obtaining information on off-line 
course development were indicated above. Information concerning the on-line 
aspects of course development regarding both the instructional programmer and 
the entry specialist will be collected on-line in real time through special software 
programming. The following information will be collected: 

(1) Interactive Instructional Programming: 

. Mean #/variability of terminals available for interactive 
authoring, testing, editing and debugging of interactive 
displays ; 

, Mean #/variability of terminals used for authoring, testing, 
editing, debugging of interactive displays; 
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. Type of interactive processing performed on interactive 
displays (initial composition, editing, etc.); 

. Mean time/variability to prepare information for input 
(finalize) during interactive authoring, testing, editing, 
debugging of instructional material; 

. Mean #/variability of terminals used for developing 
instructional logic; 

. Type of interactive processing performed on instructional 
logic (initial composition, editing, etc.); 

. Mean time/variability to prepare for input (finalize) during 
interactive authoring, testing, editing, debugging of instruc- 
tional logic, 

(2) Interactive Entry Specialist: 

. Mean #/variability of terminals available for interactive entry 
at any given time. 

3. Course Operation. Besides course development a second important evaluation 
aspect from an administrative. point of view is course operation. The areas address- 
ed in this section include: (a) a definition of the technical expertise and amount of 
in-house training required of the various personnel engaged in the operation of the 
CTS classroom; (b) a thorough scenario of the procedures utilized in the actual 
operation of the CTS classroom with sample printouts/documentation; (c) an account- 
ing of the problems encountered in the formal operration of the CTS classroom with 
recommended solutions; and, (d) a parametric assessment of the course operation 
functions to be supported by CTS. The first three areas will represent a generalized 
qualitative view of course operation; the last area, a micro -quantitative view. These 
four perspectives will provide a comprehensive description of the operational aspects 
of CTS. 

The approach to areas "a-c" above will be toward factual description of the 
personnel status and operational events as they occurred within the conduct of the 
CTS Prototype. As necessary, appropriate logging forms and procedures will be 
developed to obtain an accurate account of each of these areas. The approach to 
area above will consist of a detailed assessment of the instructional processing 
requirements of CTS. Operationally, the functional areas to be addressed herein 
are as follows: 

I. Interactive Processing 



o Student Mode 



o Instructor Mode 

o Administrative Mode 

II. Batch Processing 

o Instructor Processing 

o Adniinistrative Processing 

Each of these functional areas have a number of parameters of vital interest to 
those engaged in course development and operation which will be outlined in a 
succeeding section below. In order to render these functions more operationally 
meaningful, they will be examined within a tri-dimensional framework encompassing 
the following independent variables: 

I. Lesson Type 

o Conceptual Context 

o Functional Context 

n. Mode of Presentation 

o CAI 

o CDI 

o CMI 

in. Instructional Method 

o Tutorial 

o Drill and Practice 

o Problem Solving 
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This interpretative structure of independent variables may be graphically repre- 
sented as in Figure 10: 



Irstructional Method 




Figure 10 

Interpretative Structure for CTS Instructional 
Processing Requirements 



4, Data Collection: Procedures/Instruments (Course Operation). The dependent 
variables are the lesson parameters to be investigated (cf. below). These are 
graphically represented as cell data in the above three-dimensional figure. There- 
fore, data summations within each cell (mean's, variabilities, etc.) will be directly 
comparable between/among all levels of representation: (a) between lesson types, 
(b) among presentation modes, and (c) among instructional methods (by collapsing 
any two dimensions); and, overall, comparisons can be made among lessons them- 
selves (by collapsing all three dimensions). 

The lesson parameters of interest are given below: 

L Interactive Processing: 

A. Lesson Description: 

(a) Lesson Presentation Mode (Designation) 
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(b) # of Students Concurrently Using Lesson On-Line 

(c) # of Interactive Displays Required per Lesson 

(d) Type of Questions to be Asked (Multiple Choice/Numeric Constructed/ 
Alpha-Numeric Constructed) 

(e) Mean #/Variability of Alpha-Numeric Characters per Interactive Display 

(f) % of Interactive Displays (Graphic) 

(g) # of Static (Film) Displays Required 

(h) Amount of Audio Required (Minutes) : (If Available) 

The above parameters will serve to describe the general character of the lesson of 
instruction across all dimensional perspectives described earlier. The summation 
of this information across all lessons will describe the general character of the 
course of instruction. Disproportionate parameters v/ill be identified for follow-on 
recommendations . 

B. Student - Lesson Interaction: 

(a) # of Interactive Displays Required per Hour of Instruction (by Aptitude 
Level) 

(b) # of Static (Film) Displays Required per Hour (by Aptitude Level) 

(c) Amount of Audio (Minutes) Required per Hour of Instruction 

(d) Mean #Aariability of Student Responses per Hour of Instruction 

(e) Max. # of Dsmamic (Animated) Displays per Hour of Instruction 

(f) Max. # of Static Displays per Hour of Instruction 

(g) Max. #of Audio Messages per Hour of Instruction (If Available) 

(h) Mean Length/ Variability (Minutes) of an Audio Message (If Available) 

The above parameters will serve to describe how each lesson was utilized by the 
student across all functional dimensions described earlier. Disproportionate 
parameters will be identified for follow-on recommendations. 
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C. Testing and Inquiry: 



(a) # of Tests Adiniiiistered On-Line (Pretest, Posttests, Lesson Tests) 

(b) Mean #/VariabiIity of Test Items (by Test Type) 

(c) Mean #A^ariability of Questions Per On-Line Test 

(d) Mean #/VariabiIity of Inquiry Requests Made per Student per Hour of 
Instruction 

The above parameters will serve to define the instructional scenario in quantitative 
terms. Disproportionate parameters will be identified for follow -on recommenda- 
tions. Besides the above, a thorough description of the instructional environment 
will be given which will incl de: course description; types of questions used; man- 
ner in which student performance was managed; how tests were administered, etc. 
Examples of materials, questions, tests, branch procedures, etc., will be given 
also. 

D. Interactive Insti'uctor Mode: 

(a) WTiat Instructor Functions will be Performed Interactively (Monitoring 
of Students as Group/IndividuaUy, etc.)? 

(b) Will Instructors Interact Directly with Students Through the System? - 
To What End? How Often? For How Long on Average? 

(c) Mean #/Variability of Instructor Calls per Lesson, by Student, etc. 

The above parameters will also be moderated along the tri-dimensional structure 
described earlier. The results may serve to assist the formative evaluation during 
the course development stage; they will serve to define the instructor's operational 
role. 

E. Adnaini&lrative Inquiry Alode: 

(a) Description of the Administrative Function to be Performed Interactively 

(b) Descri|[jtioa of Administrative/Operational Reports to be Obtained Inter- 
actively 

(c) Mean ^//Variability of CTS Terminals Used for Administrative Purposes 

(d) Mean Varialjility of Administrative Liquiries per Lesson 
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As in previous sections the above parameters/queries will be fractionated along the 
tri-dinaensional structure outlined earlier. This information will serve to describe 
the general administrative role in the varied categories of computer assisted, com- 
puter directed, and computer managed instruction functions of the CTS. 

II. Batch Processing: Instructor/Administrative Modes: 

A. Description of Types of Jobs Processed: 

(a) Instructor Reports (A^ttendance Rosters/Class Profiles, etc.) 

(b) Administrative Reports (Attrition Rate/Graduation Prediction, etc.) 

B. Input Files Used: 

(a) Type (Card/Disk/Tape) 

(b) Contents (Student History, etc.) 

(c) Size (# of Records/Record Sizes, etc.) 

(d) Output Files Used: 

(1) Type 

(2) Contents 

(3) Size 

(e) Job Frequency (Daily /Weekly /etc.) 

(f) How Run? (Concurrently with Interactive Processing/Background 
Job, etc.) 

B. Formative Evaluation 

The formative evaluation of course material is the internal quality control 
phase of course development. At some point in course development, each course 
instructional programmer is faced with the problem of whether his programmed 
instructional material is suitable for operational use. The use of the reporting 
capability of the computer is intended to facilitate the instructional programmer's 
decision making problem regarding the suitability of his product. A paradigm for 
formative evaluation is as follows: 

1. Scenario. Subsequent to the development of an integral segment of CAI material 
(element/module) to the instructional programmer's preliminary satisfaction, and 
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its approbation by his peers and supervisor, a live trial run of that material will 
be conducted to determine its adequacy on a small representative sample of students. 
Through its reporting capability, the computer will provide a statistical summation 
of predetermined performance indices (by individual by group by aptitude level, etc.) 
on the representative sample. These summary reports will be obtained as a batch job 
on demand and provided to the instructional programmer. The instructional pro- 
grammer will review the diagnostic information, which will include student com- 
ments as well, and make such course revisions as are indicated. This formative 
process normally occurs prior to course implementation, but does not preclude sub- 
sequent periodic course revision during operational conditions. 

2. Data Processing/instructional Programming Requirements. In order to accom- 
plish the formative evaluation, two data processing steps having two corresponding 
instructional programmer activities are envisioned: The interface and sequence of 
data processing/course authoring procedures are as follows: 

a. Initial Data Processing: During the student trial run, a student perform- 
ance record is obtained on predetermined parameters (cf. section 3 below). This 
performance record is extracted and placed on a master tape. These performance 
records are addressable by means of an ep (enter/process) identifier which again, 
by predetermined convention, provides an index to any number of crucial variables 
(e.g. course name, student #, module #, element #, type of response, etc.). Thus, 
every student response of interest (e.g. , prover frame/lesson test responses) will 
be recorded and stored; and, upon demand request, will be retrieved and sorted. 
Initially, the following data items are processed and provided to the respective in- 
structional programmers: 

(1) All student comments (extract/print); 

(2) All unanticipated responses (extract/print). 

b. Initial Instructional Programmer Analysis: The instructional programmer 
will initially review the student comments and make instructional revisions as nec- 
essary. Further, he will review the unanticipated answer list marking them cor- 
rect/wrong as appropriate. If other answers are afforded by this review, he will 
check (X) those responses to be added to the response repertoire as alternate 
right /wrong answers. The instructional programmer will then update the course 
master tape in accordance with the indicated course revisions and the unanticipated 
responses. This initial data processing/authoring analysis is conducted either 
prior to (i.e. on short instructional segments) or in conjunction with the advanced 
course revision cycle (i.e. on relatively large and complete instructional segments), 
as discussed below, contingent on what the instructional programmer considers to 
be the most expedient for his given material. 
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c. Secondai^y Data Processing: Iii accordance with established minimum 
authoring needs and the constraints of time available for course revision, the follow- 
ing data processing reports will be made available to the instructional programmers 
and course instructors respectively: 

(1) Item Aptitude (I-A) response matrix . 

(2) Item Student (I-S) response matrix • 

A sample of each report is illustrated in Figures 11/12. 

The I-A response matrix is intended piimarily fcr the instructional program- 
mer whose main concern is with group performance (during course development) - 
on the instructional material. In contrast, the I-S response matrix is more appro- 
priate for theCTS course instructor whose main interest will be with individual 
performance under operational conditions. During the instructional development 
phase, however, both docunr.ents can be generated. Their value for purposes of 
course revision is considered to be complementary by describing the effectiveness 
of course material in terms of group (general) as well as individual (specific) per- 
formance. 

By means of the ep identifier, the instructional programmer can define which 
block(s) of instructional material he wishes to assess. This can range from a sub- 
element to the entire course of material. However, expediency suggests that an 
intermediate complete segment of material (module/several elements) will provide 
the best basis for student trials and diagnostic author appraisal. At any rate, it is 
understood that if more than one perspective of instructional material is desired 
(e.g., course, phase, module, element, subelement), a separate printout of 
Figs. 11/12 for each level is required. From a diagr.ostician^s point of view, this 
would be prohibitive. Figures 11/12 are designed, therefore, to provide the most 
desired information with the least number of reports. 

d. Secondary Ir^structional Programmer Analysis: Upon review of the I-A 
(and I-S if desired) respc^nse report, the instructional programmer will ascertain 
which frames and/or areas of the instructional material need revision. By de- 
sign the I-A reports will provide select information on student responses which 
will be organized as follows: 

. By aptitude level (hi/mid/lo) 

o Across aptitude levels (cumulative) 

. By type of frame (prover/quiz) 

« Across frames (cumulative) 

. By try (lst/2nd/3rd) 

. By instructional segment (subelement /element/module/ 

phase) 
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Item - Aptitude (I-A) Response Matrix 
(For Instructional Programmer) 
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Item-Student (I-S) Response Matrix 
(For Instructor) 
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The select information will include: 



. Percentage of correct/wrong/unanticipated responses * 

. Frequencies of correct/wrong/unanticipated responses 

• Mean (X) response latency 

. N*s for number of S^s responding 

. n*s for number of frames responded to 

. Item discrimination indices (biserial coefficients) 

Many intuitive and inferential conclusions can be drawn directly from the 
I-A response matrix, such as: 

. Which terminal performance objectives are not attained 

. Wliich aptitude level needs more attention 

. Wfiich prover frames are too: difficult /easy/invalid 

. ^^^ich elements/subelements need revision 

. Which quiz items are too: difficult/easy/invalid 

. Which lessons are not yielding sufficient mastery 

. Etc. 

Each instructional programmer's personal acquaintance with his own instruc- 
tional material will dictate the appropriate prescription for those weak/problem 
areas exposed by the I-A matrix. 

3, Basic Data Requirements. The data requirements are as follows: 

a. Background data (to be defined). 

b. Course data: 

o Course name 

. Date 

. Time of day 

. Student # 

. EP identifier (part of course/type of item/etc.) 

. R identifier (correct/wrong/unanticipated) 

. Latency time 

. Self report ( <30 words) 

. Counters (all contents) 

. Switches (all contents) 
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C. Summative Evaluation 



In consonance with the CTS Management Plan, the CTS Evaluation Division will 
conduct Phase V of the general development plan (Appendix A). As indicated in the 
plan, 'This phase will be conducted by the Product Manager concurrently with 
Phase IV (CTS operation) and is concerned with the feasibility and effectiveness of 
the entire system". With respect to training effectiveness, this requires that the 
evaluation be summative (in-breadth), as well as formative (in-depth). 

The summative evaluation is concerned with the overall effectiveness of an in- 
tegral block of course material relative to several independent criteria. In contrast 
with the formative evaluation, which represents a process evaluation directed toward 
insuring the internal reliability/validity of course material, the summative evalu- 
ation is a product evaluation oriented towards demonstrating external reliability/ 
validity of course material in relation to other presentations of the same course 
material. For decision-making purposes, the formative evaluation results will be 
primarily most useful to the in-house cadre of instructional programmers for course 
quality control purposes; whereas, the summative evaluation results will be pri- 
marily most useful to higher management personnel interested in the global aspects 
of training effectiveness for follow-on implications. The capability of the computer 
for statistical analyses will assist the evaluation division in deriving the necessary 
descriptive and inferential statistics to assess the summative adequacy of the CTS. 
Such analyses will be conducted on a background batch job basis. A paradigm for the 
summative evaluation is given below. 

1. Scenario. Three representative Army electronics courses are being considered 
for the CTS prototype evaluation. The CTS Course Development Division will pro- 
gram these three courses for CTS presentation, utilizing both CAI and CMI techni- 
ques. During the interval designated on the CTS Time Phase Plan, the CTS Evalu- 
ation Division will conduct a formal summative evaluation of the programmed ma- 
terial. In accordance with standard evaluation procedure, random representative 
samples of students will be selected from each of the three prototype courses to take 
the CTS and non-CTS instructional material. The performance of the two counter- 
part student samples will be compared on several accepted measures of achievement 
including: written and performance scores, academic setback/failure rates, and 
time to complete training. Attitudes toward CTS will be obtained from a variety of 
subgroup consumers. 

From a summative training perspective, the crucial hypothesis to be assessed 
concerns the basic reliability and validity of the CTS to do what is expected of it: 
train Army technicians to a degree equal to or better than on-going operational 
instruction, and in less time. In order to perform this assessment, the evaluation 
will specifically inquire to what extent the above dependent measures vary as a 
function of differences in two basic independent variables: training method and 
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aptitude level. As a loy -product of the statistical design, the interaction effects 
of these variables will be examined as well. Baseline statistics (means, vari- 
ances, etc) will provide the necessary parameteric and descriptive information 
regarding the performance of the two study groups; and, inferential statistics 
(F, t, x2, etc) will provide the necessary information concerning the reliability 
of any differences obtained between the respective training methods and aptitude 
levels. These findings will provide higher management with additional information 
regarding the effectiveness level of CTS and the directional emphasis to be taken 
to insure optimum follow-on implementation. 

2. Summative Evaluation Requirements. In order to perform the summative 
training evaluation, a standard evaluation format will be employed. The format 
specifications may be summarized under the following categories: 

o Objectives 

o Variables/Measures 

o Subjects 

o Instruments 

o Procedures 

o Evaluation Design 

o Decision-Making Process 

A description of aach category is given below: 

A. Objectives: The basic objectives of the summ.ative training evaluation 
(for each of the 3 CTS Pi^ototype courses respectively are as follows: 

(a) Determine baseline pai^ameters for CTS for each of the three CTS pro- 
totype courses on select training effectiveness criteria; 

(b) Compare student achievement as taught by CTS/non-CTS modes of 
instruction on select training effectiveness criteria; 

(c) Compare student achievement among 3 aptitude levels on select train- 
ing criteria; 

(d) Assess the extent of interaction between instructional methods and 
aptitude levels; 

(e) Survey sl.udenL attitudes toward CTS and relate student attitudes to 
achievement; 

(f) Determine relationship between student entry characteristics (aptitude/ 
skills) 7\ni\ in'-l.raining achievement with final training performance; 
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(g) Survey administrative personnel (instructor/staff-faculty) attitudes 
toward CTS. 

B. Variables/Measures: The basic training variables to be investigated (in each 
of the 3 CTS Prototype courses respectively ) are sub-divided into two classes, in- 
dependent nxid dependent^ as follows: 

(a) Independent Variables: 

(1) Instructional Methods (Systems) 

. CTS 

• Non-CTS 

(2) Aptitude Level: 

. High 
. Middle 

• LfOw 



(b) Dependent Variables: 

(1) Achievement Measures; 

. Module Scores 
. Phase Scores 
. Course Scores 

(2) Completion Time Measures: 

. Course #1: 
. Course #2: 
. Course #3: 



Type Criteria 
Continuous Categorical 



Writ/Perf 
Writ/Perf 
Writ/Perf 

CI 

Self-Paced 
Self-Paced 
Self- Paced 



Acad Failures 
Acad Failures 
Acad Failures 

CTS 

Self-Paced 
Self- Paced 
Self-Paced 



(3) Attitude/Opinion Measures: 



. Course Content 

. Instructional Methods /Modes 

. Student Attitudes /Opinions • Instructional Media 

. Individualization 

. Training Conditions 

. Other 

. All of above 

. Acceptance of CrS/Non-CTS 

. Instructor Attitudes /Opinions . Instructor Role (Problems) 

. Instructor -Student Relationship 

. Other 

. Acceptance of CTS/Non-CTS 

. Faculty Role (Problems) 

. Staff -Faculty Attitude/Opinions . Faculty-Instructor-Student 

Relationships 

. Required Staff-Faculty Changes 

. Other 

(c ) Matching Variable : 

It is tentatively planned that a separate regression equation (Pearson Pro- 
duct Moment) be derived for the following courses for group matching purposes: 

(1) Coarse #1 

(2) Course #2 

(3) Course #3 

The equation variables, and their raw score 'T?" coefficients, will be derived 
from a correlation matrix of the Army Classification Battery subtests and the re- 
spective course criterion measures. A modified solution to the correlation matrix 
(Wherry -Doolittle/F-max test) will be used to select the minimum number of vari- 
ables to yield the maximum multiple R^ (variance explained). The criterion variable 
will be final course achievement. 

. C. Subjects: The student samples for the summative evaluation will be selected 
at random from the normal inputs of draftees and Regular Army students to the three 
respective study courses. A minimum N of 100 per instructional method will be tar- 
geted. Similarly, a representative sample of instructors and other administrative 



support will be selected from the pool of available personnel to assess their atti- 
tudes toward CTS course development/operation/administration. 

D. Instruments: The following instruments will be employed: 

(a) Achievement Tests: In order to obtain the desired specificity in the aissess- 
ment of student achievement, in-house designed tests relating to the assigned instruc- 
tional material will be employed. This includes tests addressing both the written and 
performance aspects of training. Emphasis will be put on the use of parallel forms of 
all tests in both the study and control groups within each course to safeguard the integ- 
rity of the instruments. 

(b) Completion Time: Measures of time to complete traii^ing will be made 
available for the CTS subjects via the computer time clocking system and bookkeep- 
ing procedures as necessary; and, for the non-CTS subjects via summation of 
assigned POI hours of training plus bookkeeping of all other relevant training time 
(setback time, etc). 

(c) Attitude Questionnaires: Attitudes toward CTS will be solicited from three 
sources (students, instructors and other staff administrators). The attitudes of these 
subgroups will be measured, as appropriate, via specially prepared data gathering 
instruments and techniques. These will primarily involve the direct questionnaire 
approach. The attitude questionnaires will consist of both Likert type items as well 
as open ended items which elicit free expression of opinion and are amenable to con- 
tent analysis. 

(d) Attrition Incidence: Incidence of academic pass/fail will be obtained via 
administrative records. Separate accounting of administrative types of attrition 
will be made as well. 

(e) Achievement Prediction: An expectancy of student achievement will be 
obtained by means of empirically derived equations based on the least squares 
principle of regression. 

(f) Graduation Prediction: Both least squares regression and actuarial pre- 
diction of student success/failure and time to complete the training will be derived. 

E. General Procedures: The required statistical information on the summative 
study variables cited above will be collected in two stages: (1) baseline and (2) oper- 
ational. It is anticipated that baseline data will be collected during a period preceding 
the operational use of CTS. This set of data will represent the basic control infor- 
mation for the summative evaluation of CTS. The baseline group will represent a 
much larger pool of non-CTS subjects relative to the CTS subject pool to be obtained 
during the operational phase of the CTS Project. This will insure maximum repre- 
sentation of student participation in the two methods of instruction. As a minimum, 
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it is expected that the training conditions for the baseline control group for each 
course of instruction will be kept free of major administrative-instructional modifi- 
cations to insure the integrity of the normative baseline data. By the same token, 
intrusions into the control group training atmosphere by the evaluation personnel will 
be avoided as much as possible. The basic source of control group data will be the 
administrative records maintained by the respective study courses. 

The operational CTS data will be collected in three separate courses. It is 
anticipated that these three courses will be phased in sequentially for summative 
evaluation as they are programmed and subjected to an iterative formative evaluation 
process. In keeping with the effort to automate the CTS as much as possible, the 
summative achievement testing for the courses will be administered and scored on 
line. On the other hand, because many items will be open-ended and for other admin- 
istrative reasons, attitude measures of students, instructors and faculty will be ob- 
tained off-line. As indicated for the control group, an overriding consideration during 
the operational phase of the CTS mil be to insure, as much as possible, that the train- 
ing milieu is truly representative of a real-time operational situation, unimpeded by 
experimental or administrative intrusions and manipulations. This will minimize 
such artifacts as the Hawthorne/novelty /social approval effects, and enhance the 
generality of the results for follow -on implications. 

F. Evaluation Design: The paradigm for the summative training evaluation will 
basically consist of two independent variables and four dependent variables for each 
of the respective courses under study. The independent variables will be: training 
method (CTS vs non-CTS) and aptitude level (hi/mid/lo). The dependent variables 
will be: achievement (written/performance), time to complete instruction, attrition 
rate and attitude toward CAI. While the formal evaluation will operate -essentially 
within a quasi-experimental framework, utilizing both controlled assessment and 
independent -de pendent variable procedures, naturalistic observation techniques will 
also be employed where appropriate. 

Except for attrition and attitude measures, which reflect an ordinal level of 
scaling, all the achievement and time measures will be assumed to have attained the 
interval scaling level. In order to test the reliability of the obtained differences be- 
tween the various treatment statistics (means, variances, percentages, etc.), 
classical tests of statistical significance will be employed. Variations in design and 
scaling assumptions will dictate the use of either a parametric or non-parametric 
test of significance. Three separate types of statistical tests will be utilized relative 
to the scaling nature of the dependent variable in question and the purpose of the test: 
analysis of variance (ANOVA), t test and chi square (X^). It is anticipated that the 
ANOVA design will consist of a fixed effects model employing a two way treatment by 
levels paradigm (for achievement/time to complete measures); the t tests will be 
based on two equivalent group models; matching by pairs or group means as appro- 
priate (again, for achievement/time to complete measures); and the chi square test 
for proportions (for correlated/uncorrelated samples) will be used on the attrition 
and attitude measures. 



As the logic of the circumstances dictate, the CTS and non-CTS groups will be 
matched either pairwise on the basis of the derived matching variable (optimized pre- 
dicted achievement) or as equivalent groups on the basis of their obtained means/ 
variabilities. Further, in keeping with the emphasis noted above to maintain a realis- 
tic operational milieu, group matching will be accomplished ''after the fact*' by statis- 
tical (analysis of convariance) or random sample selection techniques as the logic of 
the situation dictates. Both of these classical matching techniques provide the most 
efficient administrative approaches to the conduct of matched group designs in an 
operational training milieu (i.e. maximum matched groups with minimum/no inter- 
ference in the on-going training process). These procedures are consistent with the 
basic emphasis of the evaluation as indicated above, to maintain the operational 
realism of all variables during the formal study period. 

G. Decision-Making Process: A statistical analysis and interpretation of the 
summative training data will be conducted upon their acquisition. The usual level 
of statistical significance (. 01/. 05) will be established as decision criteria with the 
former level offering more conclusive evidence in rejecting the null hypotheses. The 
statistical tests of significance, of course, will provide valuable information to the 
decision-making process regarding the summative training merits per se of CTS. 

From a larger perspective, however, the merit of the CTS as a viable train- 
ing system will be contingent upon a multidimensional (triangular) decision-making 
process which will concurrently take under advisement technical and cost consider- 
ations as well as its training aspects. How these three dimensions are to be weighed 
will depend on what apriori value system (assumptions) and personal probabilities are 
adopted. On the one hand, from a student end-product point of view, the tridimen- 
sional construct may best be represented by an isosceles triangle: assign more 
concern to the training aspect (long baseline) relative to the cost and technical aspects 
(short sides). However, in the "long run", the reliability and validity of the CTS as 
a viable training system will be limited both by the effectiveness of its technical com- 
ponents (hardware and software) and its cost effectiveness (i.e. approximates an 
equilateral construct). Thus, ultimately, the final value judgement of the CTS will 
be a convergent process representing an integration of divergent sources of informa- 
tion relating to its cost, technical and training effectiveness. The evaluation effort 
will endeavor to provide a fortiori empirical evidence (both pro and con) to assist in 
this decision-making process. The evaluation information of itself will not repre- 
sent the total basis for judgement but will contribute substantially to arriving at a 
realistic minimax appraisal (expected gain-loss) of CTS. 
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By -Products - Side Effects 



The CTS evaluation will endavor to be oriented not only toward predetermined 
objectives (i.e. goal-based) but attentive to significant by-products and side effects 
(goal -free) as well. These will include unanticipated aspects of CTS (both positive 
and negative) resulting from direct experience with the Prototype course development/ 
operation and system development/operation. The analysis of these operational by- 
products/side effects will be structured along the same dimensions as the basic evalu- 
ation plan itself: technical, cost and training effectiveness. Feedback of the neces- 
sary information will be accomplished by a variety of administrative instruments and 
procedures, including: CTS periodic fact sheets, logging forms (as discussed in the 
Technical and Training Effectiveness sections above), and questionnaires. The 
findings of this section will enhance the thoroughness of the evaluation and contribute 
substantially to the practicality of the CTS Prototype effort for potential follow- 
through implementation. 
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Heuristic Implications 

Besides yielding answers and solutions to giveu qaestions> it is anticipated that 
the CTS evaluation will discover, a posteriori, other critical matters relating to the 
development and operation of the Computerized Training System which will require 
further consideration and resolution. The nature and criticality of these new problems 
can, of course, only be determined through an operational run of CTS itself since it 
has no existing replicate as yet. While the CTS evaluation will endeavor to identify 
the major problems to be encountered in the real-time operation of a completely self- 
paced computerized training system, it would be presumptious to suggest that it will 
experimentally investigate and resolve all such problems in the same time frame (one 
year) as the operational run of CTS itself. Therefore, it will encumbent on the 
second generation of CTS to investigate the identified problem areas through manipu- 
lation and control of the appropriate independent variables and other standard evalu- 
ation procedures. The problem areas encountered will be classified and discussed in 
accordance with the technical and course development/operation aspects of CTS. Be- 
sides identification of potential problem areas for future investigation, the final re- 
port will endeavor to provide a generalized evaluation plan for their investigation and 
resoiution. 
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Cuini liar y/Conclusions /Recommendations 



The evaluation findings will be summarized in this section and will emphasize 
specific conclusions and recommendations. This will include a synthesis of the 
technical, cost and training results as they impact on the development and operation 
of a computerized training system and its foUow-on implementation in the U, S. Army. 
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APPENDIX A 
PRODUCT MANAGER CHARTER 
(REVISED) 

PROTOTYPE COMPUTERIZED TRAINING SYSTEM 
US ARMY TRAINING AND DOCTRINE COMMAND 
1. DESIGNATION OF PRODUCT MANAGER 

Colonel G, B. Howard is designated US Army Training and Doctiine Command 
(TRADOC) Product Manager for the Prototype Computerized Training System effec- 
tive 1 July 1973. The Product Manager reports to the Commander, US Army Train- 
ing and Doctrine Command. This charter will be reviewed annually on its anniver- 
sary date by the Product Manager to insure currency and adequacy. 

IL MISSION 

Colonel G, B. Howard is responsible for the program management of the Prototype 
Computerized Training System in accordance with DOD Directive 5000. 1, AR70-17, 
AMCR 11-16, and other pertinent regulations. 

ni. AUTHORITY 

The Product Manager is delegated full line authority by the Commander, US Army 
Training and Doctrine Command and is responsible for the planning, direction, and 
control of the allocation and utilization of all resources authorized for the execution 
I of the approved program. This includes, as applicable, definition, development, 
testing, procurement, production, distribution and logistical support. Further, he 
is responsible for assuring that planning is accomplished and implemented by the 
organizations responsible for the complementary functions of evaluation, logistics 
and maintenance support, personnel training, operational testing activation and de- 
ployment of the system. The Product Manager is supported by the offices and organi- 
zations identified in paragraph Vl.b for execution of specifically assigned tasks, 

IV. ASSIGNED RDTE PROJECTS AND TASKS 

The Army RDTE funding will be provided directly to the Product Manager through 
an Intra-Army order for reimbursable services (DA Form 2544). 



V. OTHER ASSIGNED PROGRAM TASKS OR ITEMS 



The Product Manager is responsible for overall management of: 

a. PEMA program for his assigned system, 

b. OMA as assigned, 

c. Other tasks when assigned by the Commander, US Army Training and 
Doctrine Command. 

VL INTERFACES AND PARTICIPATING ORGANIZATIONS 
a. Interfaces: 



(1 
(2 
(3 
(4 
(5 
(6 
(7 
b. 
(1 
(2 
(3 
(4 
(5 
(6 
(7 



Office of the Secretary of Defense 

Department of the Army 

US Army Training and Doctrine Command 

Human Resources Research Organization 

Department of the Navy 

Department of the Air Force 

US Marine Corps 

Participating Organizations 

US Army Training and Doctrine Commiand 

Department of the Army 

US Army Training and Doctrine Command 

Human Resources Research Organization 

Other schools and activities within TRADOC as applicable 

Contractors 

Consultants 
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VIL COMIvrUNICATIONS CHANNELS 



The Product Manager is authorized direct communications between his office, 
participating organizations and organizations with which he has interface. 

VIII. RESOURCE CONTROL 

Army resources approved to accomplish the mission will be provided to the Product 
Manager through Headquarters, US Army Training and Doctrine Command and the 
host US Army activity/installation, (See exception in para IV,) 

IX. LOCATION AND ADMINISTRATIVE SUPPORT 

a. The Product Manager's office is presently located at the US Army Signal 
Center and School, Fort Monmouth, New Jersey, Necessary facilities and adminis- 
trative support will be provided by that organization, while the Product Manager is 
a tenant at that activity. 

^ b. Upon approval of the Commander, US Army Training and Doctrine Com- 
mand, field offices may be established by the Product Manager, as required, with- 
out change of Charter, Necessary facilities and administrative support will be pro- 
vided as mutually agreed, 

X. SPECIAL EXEMPTIONS 
None 

XI. SPECIAL DELEGATIONS 
None 



APPROVED (Signed) Gruenther DATE 2 Nov 73 
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Appendix B 



Prototype Computerized Training System 
Management Plan 

1. Purpose 

The purpose of this plan is to delineate the Army command and control 
channels and procedures to be followed for the design, hardware-software develop- 
ment, course development, operation and evaluation of a protoL pe Computerized 
Training System (CTS) to be accomplished by the Office of the Product Manager. 
The responsibilities and interrelationships of the organizations and agencies parti- 
cipating in the Prototype CTS project are illustrated for guidance of all concerned. 

2. Background 

As a result of CAI Task Group Report recommendations, CON ARC directed 
the implementation of a CAI Prototype System (CONARC letter, ATIT-STM, 29 
June 1972, Subject: Computer Assisted Instruction Prototype Program Implemen- 
tation). The recommendations intiiating the prototype included the use of an inte- 
grated CAI-CMI system, use of minicomputers for the central system, and the use 
of the system for a variety of course types. 

3. Management Concept 

a. Organizations and Agencies « The following are the primary organizations 
and agencies which will contribute to the CTS Project. 

(1) Product Manager (PM), Computerized Training System, US Army Training 
._and Doctrine Command (TRADOC). 

(2) US Army Training and Doctrine Command (TRADOC) (to include TRADOC 
schools). 

(3) US Army Signal Center and School (USASCS). 

(4) US Army Southeastern Signal School (USASESS). 

(5) Chief of Research and Development (CRD). 

(6) Steering Advisory Group (SAG). 

(7) DA, Management Information Systems Directorate (DA, MISD). 
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(8) Deputy Chief of Staff for Personnel (DCSPER), 



(9) DA, Computer Systems Suppoi^t & Evaluation Command (DA, CSSEC). 

(10) Contractors: Multi Minicomputer System (MMS); Acceptance Testing; 
Evaluation; and others, as required. 

(11) Consultants. 

b. Responsibilities. 

(1) The Product Manager (PM) has the overall responsibility for the accom- 
plishment of the mission as set foii;h in the Charter; to design, develop, implement, 
and evaluate a CTS, The PM shall recommend the type of system and courses, pro- 
duce or coordinate production of the required instructional programs and manage- 
ment procedures, and supervise the implementation of the selected courses on a 
scale which will provide a valid evaluation. All coordination with other concerned 
agencies and organizations will ensue from the PM. 

(2) TRADOC will provide guidance in the form of approval of the plans and 
progress of the project as submitted thru direct channels and the SAG. TRADOC 
will provide the courses required by the CTS based upon the recommendation of the 
PM. Funding of the administrative and personnel requirements of the project will 
be provided by TRADOC through HQ, USASCS with OMA funds. Funding for the pro- 
curement of the hardware /software system will be provided for by Army RDTE and 
OMA funding as specified in paragraph IV and VIII of the Product Manager Charter. 

(3) The host activity /installation, presently HQ, USASCS, will provide admin- 
istrative and logistics support to the PM. Funds will be provided to the host 
activity/installation for the management and expenditure by the PM. TRADOC will 
authorize the personnel required by the PM through the commander of the host 
activity /installation, presently Cdr. , USASCS. TheComdt., USASESS will monitor 
and support all courses involved in the operational phase of the CTS for the PM. 
The Office of the Product Manager will be initially located at USASCS. The CTS 
prototype site will be located at USASESS for the purpose of a valid test and evalu- 
ation. 

(4) The SAG will monitor the CTS Project for the Department of the Army. 

(5) HQ, DA, MISD will review areas of interest within the CTS project and 
furnish membership on the SAG. DA, CSSEC will provide support to the PM in 
contracting, as required. 
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(6) HQ, DA, DCSPER will ensure that personnel procedures are established 
to adequately support the CTS Project. 

(7) Future contractors involved with the CTS will provide: 
(a) Hardware /soft ware system. 

(lo) Secondary display device, 
(c) Other services as required. 

(8) Consultants will be utilized as required in the CTS preparation of the final 
evaluation plan for the project o 

c. Procedures: 

The basic relationships of the PM and SAG are defined • The PM has total re- 
sponsibility and control of the CTS Project based upon directives from DA and 
TRADOC. The SAG is the reviewing and coordination group which monitors the 
project for DA. 

4. DA Steering Advisoi^y Group 

a. The Group will meet at a time and place designated by the Chairman. 
The Product Manager as Executive Secretary of the Group is responsible for pre- 
paring a draft agenda, notifying members of meetings, preparing and distributing 
minutes of meetings, and insuring that actions required as a result of meetings are 

b. The functions of the SAG are: 

(1) Monitoring the progress of the Prototype CTS Project for the Department 
of the Army. 

(2) Providing a mechanism for coordination, exchange of information and 
review of the Prototype CTS Project by all interested parties. 

(3) Exercising control over functional requirements and technical character- 
istics of the prototype CTS. 

c. Membership on the Steering Advisory Group is as follows: 

(1) Director of Army Research - Chairman 

(2) Prototype CTS Product Manager - Executive Secretary 
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(3) DA, MISD 



(4) 


DA, DCS PER 


(5) 


DA, OCRD 


(6) 


TRADOC 


(7) 


USASCS 


(8) 


USASESS 



5. General Development Plan 

a. This project is divided into five functional phases as outlined below: 

Phase I - System Specification. This phase has been accomplished by the PM. 
Tb-i phase ran through 18 April 1973 when the Request for Proposals (RFP) was 
released. 

Phase II - System Development. This is to be accomplished by a future con- 
tractor and will end eighteen months after contract award. Delivery will be phased 
as specified in the RFP. 

Phase III - Course Development. This phase will run concurrently with 
Phase I and II and will be accomplished by the PM. 

Phase IV - CTS Operation. This phase is concerned with the preliminary test- 
ing and evaluation of the prototype CTS to include student trials. The operation of 
the one year prototype test and evaluation of the CTS and student performance will 
follow. This phase commences in the thirty -third month and runs through the forty- 
fifth. 

Phase V - CTS Evaluation. This phase will be conducted by the PM concur- 
rently with Phase IV and is concerned with the feasibility and effectiveness of the 
entire system. This phase will start in month thirty -three and conclude in month 
forty -eight. A final repoi^t will be prepared at the conclusion of the evaluation phase 
to include recommendations for future actions. Consultants/contractors will be 
utilized in evaluation as appropriate. 

b. The time frames presented in this plan are required for project comple- 
tion to meet a 1 August 197G deadline. 
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TAB 1 to Appendix D: Operational CTS Time Phase Plan (Preliminary) 



1. General. The process of life-cycle costing is predicated upon the time phasing 
of events throughout the life of the system. Periods of amortization, discounting 
techniques ♦ recurring costs are all dependent upon when the cost occurs and for how 
long. Accordingly, estimates will be made concerning the time phasing of major life- 
cycle costs (developmental, investment, operational) based upon data obtained during 
the CTS prototype testing • 

2. Time Phase Activities. The activities that must be accomplished as part of the 
operational life of the CTS are shown in figure Dl-1. As this is a preliminary plan, 
exact timing of activities are not known. The exact timing and critical path relation- 
ships will be determined after the prototype testing period. The specific activities 
anticipated at this time are as follows: 

(a) Acquisition Process: 

(1) GFSR Approval 

(2) Staff Organization 

(3) Develop DFSR 

(4) DFSR Approval 

(5) Product Master Plan Dev & Approval 

(6) Prepare System Specifications 

(7) RFP and Vendor Reply 

(8) Evaluate Proposals 

(9) Contract Negotiations 

(10) System Delivery and Installation 

(b) Manpower Planning: 

(11) Initial Manpower Plan 

(12) Organize and Finalize Plan 

(13) Acquire Initial Staff 

(14) Acquire Additional Staff 

(c) Facilities: 

(15) Determine Plant Requirements 

(16) Approve and Coordinate New Construction 

(17) Plant Remodeling 

(18) Classroom and Outside Plant Installation and Test 
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(d) Course Development: 



(19) Specialized Training for Authors 

■20) Structure Course Content 

(21) Develop Off-line Course Material 

(22) Off-line Coding of Lessons 

(23) Finalize Lessons 

(24) Enter and Debug Lessons 

(25) Student Trials and Debugging 

(e) Operation: 

(26) Operation with Course Evaluation and Updating 

3* Summary. Based upon the above events and when they occur the time phase 
plan will become input data for the operational CTS Cost-Effective Model (TAB 2). 
The time phase plan will also provide a time frame for the life-cycle operational 
CTS for future planning consideration. 
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TAB 2 to Appendix D: Operational CTS Cust-Effectivc Model 



1. General. A CONARC Task Group, in their April 1972 report on Computer 
Assisted Instruction, developed a computerized cost model tor |)re(licting opera- 
tional life-cycle costs for CI and CAI at selected CONARC scliools. The model was 
programmed and run at Fort Monmouth, NJ and the results included in the Tusk (li'oup 
Report. For purposes of predicting the life-cycle cost of an opeiational CTS an<l 
comparing it to existing CI costs, the model has been reviewed, uprlated and will pro- 
vide the basis for the CTS cost-effective model. 

2. Assumptions: 

a. The results of the CTS prototype project concerning anticipated percent 
reduction in training time, percent utilization of student terminals, and equal or 
better student effectiveness remain valid for technical training tliroughout TRADOC. 

b. The flow of students into the courses will be evenly distributed throughout 
the year so that the number of student terminals will not have to be increased to 
handle peak student loads, nor will student terminals be idle during the slack periods. 

c. Yearly student input will remain constant for the life-cycle of the CTS 
operational system. 

d. Funds will be available to acquire all required equipment and personnel at 
the proper time. 

e. Students will be effectively utilized upon completion of their self-paced 
course. 

f. The operational life of a CTS is 10 years. 

3. Inputs: 

a. Data required by the model will be provided by selected TRADOC schools 
concerning general school data and specific course data. The remaining data will be 
provided by the CTS evaluation team and will be based upon results of the CTS proto- 
type testing. 

b. Data items by type and origin are as follows: 
A. External: 

(a) School Data: 
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(1) Avex'age Classroom Size 

(2) Classroom Remodeling Cost 

(3) Computer Room Remodeling Cost 

(4) Average Distance from Computer to Terminal 

(5) Administrative Area 

(6) Administrative Remodeling Cost 

(7) Cost of Any New Buildings o.r Existing Amortization 

(8) Utility and Maintenance Factor for Buildings 

(9) Civilian Staff, Existing 
(10) Military Staffs Existing 



(b) Course Data: 



(11) Student Input 

(12) Modal Grade 

(13) POI Hrs per Course 

(14) Hours of Course Development per POI Hours 

(15) Number of Shifts per Day 

(16) Number of Hours per Shift 

(17) Number of Days per Week 

(18) Number of Weeks per Year 

(19) Modal Grade of Course Authors 

(20) Modal Grade of Course Instructors 

(21) Number of Course Authors Required 

(22) Number of Instructors Required 

(23) Night Shift Operation 

(24) Training Equipment Costs 

(25) Training Equipment Maintenance Factor 

(26) Training Aids Cost 

(27) Course Attrition and Recycle Rates 

(28) Number of PE Hours per Course 



B. Internal: 



(a) Prototype: 

(29) terminal Utilization 

(30) Reduction in Training Time Due to CTS 

(31) Reduction in Attrition Due to CTS 

(32) Time Phase Plan 

(33) Carrel Cost 

(34) Terminal Cost 

(35) Hardware/Software Cost 

(36) Terminal Maintenance Cost Factor 
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(37) Hardware/Software Maintenance Cost Factor 

(38) Ratio of Staff Terminals/Student Terminals 

(39) Carrel Size 

(40) Cost per Ft of Outside Plant (CPU to Terminals) 

(41) Author Hours Required per CAI/CDI Hour of Course Development 

(42) " " " " CMI " " '* '* 

(43) Support " " CAI/CDI 

(44) " " " " CMI 

(45) Optimal Number of Instructors per 50 Terminals 

(46) Optimal Number of CMI Instructors per 50 Terminals 

(47) Military Staff Requirements 

(48) Civilian Staff Requirements 

(b) Other: 

(49) Civilian Pay Table 

(50) Military Pay Table 

(51) Present Value Table 

4. Cost Estimating Relationships: 

a. A Cost Estimating Relationship (CER) is a function of various input and other 
CER. Figure D2-1, CTS and CI cost flow, indicates the input of data items from 
paragraph 3 and the flow through various CER terminating in major subsystems as 
indicated by capital letters. 

b. The following lists the CER for CTS and CI operational costs. The letters of 
Column A, B and C refer to Figure D2-1, and the numbers of Column B and C refer 
to the input data listed in paragraph 3. 



A • B« C . 

Factors to be as a Inputs and 

Computed function of CTS Other CER CI 



a. (b.,12,50) same 

b. (11,13,27,30,31) same 

c. (6,7) (7) 

d. (11,13,24,30) (24) 

e. (13,28,32,41,42) (13,14,28) 

f. (13,28,32,43,44) (13,14,28) 

g. (5,8) same 

h. (e., 19, 49, 50) same 
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3-8 a Inputs and 






luuciion 01 o 1 o otner o 


CI 


1 • 


/f oc ^1^7 AO. .ici 

(I • , <iO, 4 ( , 4o, 4y, OU) 


same 


j* 


(a, ^0) 


• same 




(b, lo, 16, 17, 18, ^y, Jo) 


XT / A 


1 

L • 


(K,y, JLU, io,-:o,0ii,4f,4o) 


^D, , y , iu; 


m. 


(1. ,49, 50) 


same 


n. 


(K, 2o , 4o, 4d) 


(b. , 11) 


0 • 


/I - /I Q /I \ 

(K. , 4, J4) 


M /A 

IN /A 


P* 


(K., 3J) 


M / a 

In/A 




(r • , io, ^C5) 


same 


r s 


/It 1 OQ QQ\ 




D • 


. , Of) 


n/a 


t « 


(r, ,q, , 2, 7) 


(r, ,q. , 7) 




(v., 36) 


\T / A 


V • 


(r • , q, , i, cJ; 


same 


w. 


(n » , 4y , 0 u ) 


same 


TT 
U . 


(a, , m, , w) 


same 


V , 


(0. , p, ) 


M /A 


w. 


(k.,35) 


N/A 


X. 


(e.,t.,3) 


(c.t.) 


Y. 


(g,,j.,s.,u.,v.) 


(g..j-.v.) 


Z. 


(d.,h.,i.) 


same 



5, Outputo The operational CTSCost Effective model will perform the calculations 
as indicated above and then summarize the data for output. The output will include 
life-cycle costs for each course, school and representative TRADOC operational 
system for a CTS and CI environment. Based on the time phase plan of TAB 1, the 
model will include transition cost from CI to CTS, total system cost discounted 
yearly at an appropriate discount rate, break even points, and cost per POI hour. 
Where appropriate and practical, a sensitivity analysis will be made concerning items 
such as varying student input, discount rates, etc. 

6. Summary. The operational CTS Cost Effective Model provides information rela- 
tive to TRADOC decisions concerning what courses and schools will have a CTS cap- 
Pibility, and to what extent, based on cost effectiveness nd a realistic estimate of 
the life-cycle cost for an operational TRADOC CTS. 
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Glossary 



This glossary is provided to define the use of terms within the context of this 
report. 



o CTS: Computerized Training System. The integration of the computer into 
a totally self -paced trgcining system. In a CTS, the computer serves 
as a teaching medium (via its peripheral display terminals), a surro- 
gate instructor, a classroom management tool, and as well as per- 
forming many school administrative functions associated with training. 



o CAI: Computer Assisted Instructionc The use of the computer as a multiple 
instructional mode teaching medium, functioning interactively with the 
student, providing him with lesson material and evaluating his inter- 
action with the lesson material. The lesson material and teaching 
logic is stored within the computer memory. 

o GDI: Computer Directed Instruction. The interactive use of the computer 
as an adjunct to and a director of other media of instruction. In this 
mode, the computer is used to interact periodically to check the stu- 
dent's progress and provide remediation of needed, and further dir- 
ections on how to proceed. 



o CMI: Computer Managed Instruction. The use of the computer as a class- 
room management tool. In this mode, the computer is used to grade 
tests, prescribe remedial work, prescribe lessons to be studied, 
designate madia to be used, schedule equipment and media, and 
monitor student progress . 

o Instructional Method: General method by which instructional material is 
organized and developed for teaching to include: tutorial, drill and 
practice, simulation and gaming, problem solving and others. 

o Presentation Mode: Specific mode of presenting/conducting instructional 
material to include: CAI, GDI, CMI (defined above). 

o Lesson Type: General characteristic of instructional material indicating 
its theoretical versus practical orientation to include: conceptual 
and functional lesson types. 

o Instructional Model: A specific set of instructional strategies (cf . below) 
structure the interactive process between the individual student and 
the subject matter. 
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Instructional Strategies: Decision mechanism that allows for selection from 
alternative plans of instruction the one that hopefully will lead to an 
optimal performance level. This decision mechanism involves the 
characteristics of the learner, the structure of the cui^riculum material 
being developed, the behavioral processes being utilized by the student, 
as well as the student^i- coping behavior that results in maximizing 
his rewards and minimizing his efforts. 

Instructional Programmer: The individual responsible for developing lessons 
for CTS. The instructional programmer's duties under the ''one man 
concept" in a CTS encompass the following: authoring of lesson text, 
both for on and off-line presentation; preparation of the computer 
coding essential to the execution of the on-line lesson material; testing, 
debugging and editing the lessons; and design and development of 
associated graphics and training devices. 

Display: The presentation of text and graphics on the display device using 
contiguous display commands which is limited to a single screen pre- 
sentation. 

Frame: A component in the instructional process. A frame will contain one 
or more displays, e.g,, a Skill Frame will consist of a message dis- 
play, a question display, and the remediation and reinforcement dis- 
plays . 

Student Tern7.h;ai: A configuration of input and output devices and their 

^/A^ironment which have been human-engineered for use by a student 
in a CAI/CMI application. 

Technical Training: A course which results in the award of a Military 

Occupational Specialty (MOS) and/or an Additional Skill Identification 
(ASI) in which the major portion of training is devoted to the develop- 
ment of hard skills associated with the installation and maintenance 
of equipment. 
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